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Multimedia Information," filed November 19, 2001. 



25 BACKGROUND OF THE INVENTION 

[02] The present invention relates generally to processing an information stream and 
more particularly to presenting portions of an information stream. 
[03] It is oftentimes desirable to be able to index an information stream and then later 
to access the information stream based on the indexing activity. Consider a concrete 

30 example where the setting is a marketing research effort. 

[04] Sociologists engaging in market research often observe focus groups to 
understand behavior. They take written notes as a complement to a video recording of a 



session. Conventionally, moderators study the behavior of the focus group. The 
moderators may synchronize their watches to the video time so that they can correlate 
their observations and notes with the video recording when they review the video. 
[05] hi one scenario, a marketing researcher could observe members of a focus group 
5 imwrapping a prototype package. During the session, the focus group moderator records 
video of the participants' actions while unwrapping the package, making notes of 
important events and flagging them for later review. For example, the moderator may 
expect to see certain events such as "uses scissors," "removes foil," and so on. Each time, 
a member of the focus group "uses scissors" to open the package, one or more moderators 

10 will note the time of the activity in her notebook. If a member of the focus group 
"removes foil," the time of occurrence of that activity is noted in a notebook. 
[06] A subsequent review of the video recording and the moderators' notes may 
require substantial effort to collate and organize the information. This effort can be 
especially tedious if it is also necessary to correlate the noted events recorded in the 

1 5 various notebooks. This might require searching through the potentially many hours of 
one or more video recordings to obtain snippets of video corresponding to the times of the 
events noted in the notebooks from many moderators. 

[07] A need therefore arises to facilitate the presentation of recorded information. 
There is a need for the ability to identify noted events in the recorded information and the 
20 ability to access the recorded information at the times corresponding to the noted events. 

SUMMARY OF THE INVENTION 
[08] In accordance with various aspects of the present invention, an information stream 
(media stream) can be timestamped with event markers to indicate points of interest in the 
25 information stream. Repeated occurrences of an interesting event in the information 

stream can be timestamped with the same marker. The timestamped information stream 
can be presented on a suitable medium where segments of the information stream having 
the same marker can be grouped together to facilitate subsequent review of the 
information stream. 

30 
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BRffiF DESCRIPTION OF THE DRAWINGS 
[09] An understanding of the present invention is provided by the following figures 
along with the discussion that follows, where: 

Fig. 1 is a high level general block diagram illustrating aspects of the 
present invention; 

Fig. 1 A shows an aspect of the present invention in which plural 
information streams and/or pl\u:al event marker streams can be produced; 

Fig. 2 is a general block diagram of an illustrative embodiment of the 
present invention; 

Fig. 3 illustrates timestamping of an information stream; 

Fig. 4 illustrates various embodiments for a paper presentation; and 

Fig. 5 is a flowchart highlighting the steps of a particular implementation 
of the present invention. 

DESCRIPTION OF THE SPECIFIC EMBODIMENTS 
[10] Fig. 1 is a high-level block diagram of an event marking system, illustrating 
various aspects of the present invention. An information recorder 102 produces a 
recording of "something." The "something" might be an audio-visual recording; e.g., a 
recording of the activity of a group of people interacting with each other. The recording 
can be a series of images. One might record the output of a sensor; for example, force 
transducers in a processing plant might be used to monitor pressure, load forces, and the 
like to produce a series of sensor readings that can be recorded. The information stream 
therefore can be any form of data such as audio, video, raw data (sensor data), or any 
other time-based media. 

[11] As alluded to above, the information recorder 102 produces an information 
(media) stream. In the case of an audio-visual recording or the like, the information 
stream would be a continuous stream. Alternatively, the information stream may 
comprise discrete information, as in the case for example of a digital camera producing a 
series of images. In the sensor example, the information stream can be a continuous 
recording of the instantaneous reading of the sensor(s), or the information stream can be 
discrete data representative of readings taken at discrete time intervals. 
[12] In the discussion which follows, the term "information" will be used to refer to the 
recorded stream of information being processed, whether digital information or analog 
information. A common information stream is a media (multi-media) steam comprising 



audio-visual information. However, the present invention is equally applicable to streams 
of information other than audio or visual data. As mentioned above, sensor data can be 
recorded over a period of time for monitoring purposes. While not normally associated 
with multi-media streams, such data can be treated as a data stream that can be 
S timestamped and processed in accordance with the present invention. 

[13] An event marker 1 04 is used by a user to signify the occurrence of events. When 
the user observes a noteworthy event, she will act in a predetermined manner to trigger 
the event marker ("marking the event")- hi response, the event marker marks the 
occurrence of the triggering action along with suitable timing information to produce 

10 timestamped markers, indicating what event occurred and when the event occurred. In 
the case of multiple recording devices, information indicative of the particular recording 
device with which the event marker is associated can be included in the timestamped 
markers. As will be discussed in more detail below, one such trigger action is the 
scanning of a barcode. Other triggering actions might include the user speaking a phrase, 

1 5 making a selection of a computer graphic on a visual display or otherwise making some 
detectable gesture. According to an aspect of the invention, repeat occurrences of a 
noteworthy event are marked by the same triggering action. For example, each time a 
person enters a room, that event can be marked by scanning the same barcode, or by 
speaking the same phrase, and so on. 

20 [141 The trigger action can be noted by recording a representation of the specific 

trigger action (e.g., a barcode is represented by a number), and recording a time index or 
some other suitable timing information of the occurrence of the triggering event. The 
timing information 1 16 can be a timing source that is common to the information recorder 
102 and to the event marker 104. Alternatively, the timing information can be derived by 

25 the information recorder and by the event marker independently of each other; for 

example, each device may have its own clock. The event marker 104 can be configured 
to record the timestamped markers (timestamps) in storage provided within the device, 
and thus requires no interaction with the information recorder during recording. 
Alternatively, the event marker can be configured to communicate the timestamps to the 

30 information recorder over a communication link 114 or to a remote storage component 
(not shown), if the event marker is not equipped with suitable storage. 
[15] A presentation engine 106 receives fee information stream 1 12a fi-om the 
information recorder 102 and a set of timestamps 1 12b fi-om the event marker 104 (or 
fi-om a storage component). The presentation engine synchronizes the information stream 



with the timestamps to correlate each triggering event with a portion of the information 
stream. The presentation engine then generates a suitable presentation 132 of one or 
more portions of the information stream, as identified by the timestamps. Alternatively, 
the event marker 104 can store the timestamp information in a storage component 108; 
5 e.g., in a timestamp file or the like. The presentation engine can then access the 
timestamp information from the timestamp file. 

[161 The presentation 132 can be formed on a printable medium. Alternatively, the 
presentation can be made on a visual display device, such as on a computer terminal or on 
a handheld device. The presentation can be virtual; e.g., a virtual reality device such as a 

10 heads-up display can be used to project the presentation. 

[17] Fig. 1 A illustrates another aspect of the invention wherein more than one 
information recorder 102' can be provided. In this configuration, the information stream 
1 1 2a' will comprise one or more separate streams, representing information streams from 
each of the information recorders. Alternatively, some information recorders might be 

1 5 configured to combine their outputs to produce a single stream. 

[18] Fig. 1 A also shows a fiirther aspect of the invention in which one or more users 
using one or more event markers 104' can trigger events to be recorded. The flow 1 12b' 
represents one or more streams of timestamped markers from the one or more event 
markers. As will be discussed in fiirther detail, the timestamped markers may include 

20 additional information to identify the specific event marker 104' that produced the 
timestamped marker. 

[19] Fig. 2 is an illustrative embodiment of an event-marking system, showing an 
example of a specific implementation. A recorder 202 can be a digital recorder that 
provides a continuous information stream 212a to a multi-fiinction peripheral (MFP) 206. 
25 Alternatively, the recorder can be a digital camera that feeds a discrete information stream 
of images to the MFP. 

[20] hi accordance with the specific implementation shown in Fig. 2, events can be 
marked using a barcode reader 204a to scan in barcodes 204b to produce a set of 
timestamped markers. First, the user is provided with a sheet of pre-printed barcodes 
30 204b, which are unique and are not associated a priori with any event or context; the 
barcodes are initially "blank." 

[21] When the user is ready, she can assign event names to each barcode in accordance 
with a given particular use. For example, an observer in a product testing group for a new 
package may be interested in studying events such as "person used scissors to open the 



package," or "person removed the foil from the package." She could assign each event to 
a barcode 214a, 214b, as shown in Fig. 2. As shown in the figure, the observer can leave 
some barcodes unassigned. 

[22] During the observation session, participants' activities will be recorded with a 
5 suitable recording device(s) to create a recording of the session. Whenever the observer 
notices someone using scissors to open a package, she can scan the barcode 214a 
associated with that event to indicate its occurrence by marking the event. Similarly, the 
occurrence of someone lifting a foil from the package can be noted by scanning the 
barcode 214b associated the "removed the foil" event. In this way, the information 

10 stream (here, a recorded session) can in effect be timestamped with barcodes. In 

accordance with an aspect of the invention, repeated occurrences of an event can be noted 
by repeated scannings of the barcode associated with that event. Consequently, a given 
barcode may very well appear more than once in the timestamped information stream. 
[23] The scanned barcodes are converted to a data stream and can be stored in the 

15 barcode reader 204a, along with suitable timing information. The data stream along with 
the timing information can be later communicated to a data collection station. 
Alternatively, the data can be communicated as it is being collected. If a particular 
barcode reader lacks adequate long term storage capacity, the data stream and any needed 
timing information can be communicated (e.g., wirelessly) to a separate data collection 

20 station (not shown). As will be discussed below, the timing information can be relative or 
absolute timing information. 

[24] In accordance with an aspect of the invention, the meaning of a barcode need not 
be provided in advance of the observation session. For example, the observer may 
witness an unexpected behavior such as someone opening a bag using his teeth. The 

25 observer can scan one of the vmassigned barcodes 216 upon seeing the imexpected 

behavior, and from that point on scan the same barcode each time a "uses teeth" behavior 
is observed. The observer can mark the barcode sheet 204a with the event name "teeth" 
as shown in the figure to identify the new behavior. Further detail and variations of this 
aspect of the invention will be discussed below. 

30 [25] The MFP 206 receives the information stream 212a from the recording device 
and, likewise, receives the scanned barcodes 212b that were collected by the barcode 
reader 204a. The MFP processes the information stream to "timestamp" the information 
stream with the barcodes. The scanned barcodes represent a specific example of 
"timestamped markers." The act of associating the scarmed barcodes with points in time 



in the information stream represents a specific example of "timestamping" the 
information stream. Further detail and variations of this aspect of the invention will be 
discussed below. 

[26] When the barcode sheet 204b is fed into the MFP, the MFP can produce a 
5 presentation 232 of the timestamped information stream. For example, the images shown 
in the example of Fig. 2 represent the timestamped markers that were collected during the 
study session. In the particular implementation shown in Fig. 2, the presentation 232 can 
be a "video paper" output. Video paper is a technology for presenting audio-visual 
information on a printable medium such as paper. This technology is discussed in more 
10 detail in one or more of the above-referenced co-pending applications. The video paper 
presentation 232 includes representative thumbnail images 252a, 252b of portions of the 
information stream that have been timestamped. Those portions of the information 
stream that are timestamped with the same barcode can be "grouped" together on the 
video paper. 

1 5 [27] The arrangement of the groups can be directed by any of a number of ways. Li 
one case, a simple grouping with some automatic layout can be used, where the 
arrangement is made absent any input of arrangement information from a user. For 
example, a simple listing that is not specific to the particular collection of barcodes. In 
another case, an electronic representation of the format of the document that contained the 

20 original barcodes can be used to direct the formatting of the generated the output. This 
electronic format might include explicit space for insertion of the clips. In yet another 
case, a paper prepared by a user containing the barcodes along with other marks on the 
paper can be used to direct the arrangement. The paper can be scanned and a new version 
of the scanned image is generated with the clips inserted depending on the location of the 

25 barcodes in the paper. 

[28] In the case of an audio-visual information stream, a representative thumbnail 
image might be a frame of video. Thus, the images 262 relating to the event "uses 
scissors" are shown grouped together. Similarly, the images 264 relating to the event 
"removes foil" are shown grouped together. Likewise, the images 266 relating to the 

30 unexpected event "teeth" are shown grouped together. Recall that although the "uses 
teeth" event was unexpected, the invention nonetheless allows the user to identify 
occurrences of the unexpected event in the information stream, and to later group together 
occurrences of the unexpected event. 
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[29] The labels appearing in the video paper that are used to identify the groups can be 
taken from the barcode sheet 204b. The MFP 206 can perform simple "cut" operations 
on predefined locations of a scanned-in image of the barcode sheet to obtain segments 
containing the user-provided event names, including the names of those unexpected 
5 events. Subsequent "paste" operations can be performed on the video paper image to 
provide the group labels in the video paper, as exemplified in the figure. It can be 
appreciated that optical character recognition (OCR) processing can be applied to extract 
text written in the predefined locations. However, the cut-and-paste approach would 
preserve any non-textual markings that the user might make in addition to the text. 

1 0 [30] For example, in accordance with an implementation of the cut-and-paste 

technique, the image of the scanned barcode sheet (with its user-provided markings) can 
be compared with an electronic image of the sheet as originally printed, using well-known 
matching algorithms. Any regions containing differences could be extracted and placed 
on the sheet to be printed, the differences being the user-made markings. 

1 5 [31] hi an altemative implementation, an output image (e.g., video paper image) can be 
produced based on the scanned-in image of the barcode sheet as marked up by the user. 
The MFP 206 can locate each barcode in the scanned barcode sheet and immediately 
below each barcode insert sufficient space in the output image to accommodate all the 
clips associated with that barcode. The remaining portion of the image of the scanned 

20 barcode sheet would be "pushed down" to make this space. New flow, such as 

repositioning page breaks, could be used as appropriate to format the output image. 
[32] A feature of the video paper 232 is that the user can subsequently access portions 
of the information stream represented by the thumbnail images. This constitutes a 
playback mode of operation. Briefly, each thumbnail image formed on the video paper 

25 has associated with it a graphical indicium, such as a barcode 254. When a user scans the 
barcode, the barcode id is transmitted to a playback device (not shown). The playback 
device can then access the information stream and playback the portion of the information 
stream associated with the thumbnail image. Additional detail about this aspect of video 
paper can be found in one or more of the above-referenced applications. It can be 

30 appreciated that the video paper component of this particular embodiment of the 
invention can be used to create, store, and manage additional information to further 
facilitate access to the information stream. 

[33] Fig. 2 shows that the presentation can be made on a video display device 234. It 
can be appreciated that a suitable user interface can be provided to allow a user to access 

8 



those portions of the information stream that are timestamped with the same barcode. For 
example, a display much like the image appearing on the video paper 232 can be 
presented on a video display device. Using an appropriate input device, a user can select 
a thumbnail image and view the corresponding portion of the information stream. For 
5 example, a ten second snippet of the information stream can be played back. 

Alternatively, the user interface might allow playback of the information stream starting 
from a location determined based on the timestamp associated with the selected 
thumbnail image. These and other playback aspects of the invention will be discussed in 
further detail below. 

10 

Event Markers 

[34] Steps for timestamping an information stream in accordance with an embodiment 
of the present invention are highlighted in Fig. 5. It will be appreciated from the 
discussion which follows that some steps need not be performed in the order shown in the 
15 flowchart. The steps are merely laid out in a flow chart format for convenience of the 
discussion. 

[35] Generally, a set of event markers is provided on a sheet of event markers, as a 
source of blank markers. In a step 502 of the particular embodiment of this aspect of the 
invention, the event markers are a set of unique pre-printed barcodes. The pre-printed 

20 barcodes can be manufactured and distributed to retailers, who then sell them to end 
users. The barcodes can be provided as pre-printed sheets of paper 204b (Fig. 2), 
provided on a roll of stickers, provided in a notebook format, and so on. 
[36] As mentioned above, barcodes are used in a specific implementation of this aspect 
of the invention, and an information stream is "timestamped" to mark events of interest 

25 by scanning in barcodes corresponding to those events. However, as will be discussed 
shortly, other forms of markers are possible. 

[37] In particular embodiments of the invention, the barcodes will be represented in a 
form suitable for the underlying technology (step 504). Thus, in the case of a software- 
based embodiment, the markers are likely to be represented by the software as unique 
30 numbers (marker ID); e.g., N-bit data. 

[38] One reason that uniquely identified markers might be preferable is to avoid 
collisions. For example, if the same scanning device is used in two different projects 
there is no danger of confusing the timestamped events of project A with those of project 
B. Also, additional markers can be added at any time without concern for collisions with 



existing markers. The use of unique markers allows for an unlimited ID space, which 
would not be possible if the markers were tied to an input device. For example, 
generating markers which correspond to the buttons of a 3-button mouse would limit the 
ID space to three different markers, and thus three events. A full keyboard could provide 
5 about 100 unique markers, though somewhat unwieldy to use and still quite limited as to 
number of events. 

[39] A second reason to use uniquely identifiable barcodes is that they can be used to 
associate specific pieces of paper with event markers. For example, the ability to tie a 
specific police officer's investigative notebook page with an event marker can be useful 

10 for establishing the presence of the officer at a particular event. If several markers were 
placed on each page of such a notebook, then their presence in context provides strong 
evidence that the officer was indeed the person who marked a particular event. Since the 
marks and barcodes are "entangled" with real world events about which testimony can be 
obtained, they provide a stronger chain of evidence that a specific person marked specific 

15 events. 

[40] It can be appreciated that uniquely identifiable "markers" other than barcodes can 
be used to timestamp an information stream. For example, the user can be provided with 
a visual interface on a video device such as a tablet PC, for example. The visual interface 
might have a set of graphics such as a set of checkboxes. Each checkbox can be uniquely 

20 identified. A user can associate some of those checkboxes with specific events particular 
to the observations being studied; e.g. "gate opens," "person enters room," "rat eats 
cereal," "rat eats cheese," and so on. For example, the interface can provide an input 
screen which allows a user to enter descriptive information for an event that can be 
associated with a checkbox. 

25 [41] Each time an event is observed, the corresponding checkbox can be "selected" 
using a suitable input device, thus marking the event. The interface can then record the 
specific identity of the checkbox and associate timing information indicating when that 
particular checkbox was selected, thus creating a timestamped marker. This information 
provides a basis for timestamping a corresponding information stream with the checkbox 

30 identifiers. As with the barcodes, the user can leave some checkboxes unassigned. The 
unassigned checkboxes can then be assigned on the fly when unexpected events occur. 
[42] In the foregoing embodiments, a set of predetermined markers were provided to 
the user. For example, pre-printed barcodes had to be provided, or checkboxes on a 
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graphical interface were presented to the user. These predetermined markers serve as 
visual cues for making an association with an event. 

[43] In accordance with an alternative embodiment of this aspect of the invention, a 
voice-based event marker can be used to timestamp an information stream. This 
5 particular embodiment of the invention obviates the use of pre-defined, pre-printed 
markers; the markers can be defined on the fly, during the observation session. The 
observer simply speaks a particular phrase when a certain behavior (event) occurs; e.g., 
"uses scissors," "lifts foil," "teeth!" and so on. Suitable voice recognition processing can 
be applied to capture the speech, analyze it, and associate a imique identifier with the 
10 recognized speech. Thus, each time the user speaks "uses scissors," the voice recognition 
process can produce and store the same unique identifier along with any needed 
timestamping related information. In this way, the information stream can be 
timestamped with the spoken observations. 

[44] A further aspect of the present invention is the encoding of associated information 
1 5 along with a timestamped marker when an event is marked. Thus, in addition to storing 
an identifier (marker ID) that uniquely identifies the marker (e.g., barcode) when an event 
is marked, a device identifier (device ID) which indicates the specific recording device 
can be associated with the marker ID. According to one implementation, the device 
identifier can be determined by scanning a barcode that is associated with the recording 
20 device. The device identifying barcode can be formed on the recording device itself the 
user simply scans the barcode on the recording device. The device identifying barcode 
can be printed on an operator's manual associated with the recording device, or it can be 
printed on a sheet of barcodes that is associated with the recording device. According to 
another implementation, the device identifying barcode can be manually entered as a 
25 sequence of key presses made on the scanning device. It can be appreciated that still 
other implementations are possible. For example, in accordance with another 
implantation, a device identifier can be programmed into the recording device at the 
factory. 

(451 Storing the recording device identifier creates an association between the 
30 timestamped markers (timestamps) and the recording device. This aspect of the invention 
is especially usefial where multiple sources of information streams and multiple sets of 
timestamps may be presented to an MFP 206 device. The MFP (or other similar device) 
can access the proper information stream based on the device ID that is associated with a 
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given set of timestamps, without requiring the user to manually indicate which 
information streams go with which set of timestamps. 

[461 Other associated information can include information which identifies the user. 
This allows the MFP to access portions of the information stream(s) that were 
5 timestamped by a particular user. It can be appreciated that still other useful information 
can be associated with a timestamp. 

[47] Fig. 4 shows that annotations can be associated with each event marker. Thus, in 
a particular embodiment, a barcode sheet 404b similar to the barcode sheet 204b in Fig. 2 
can contain annotations 412 made by the observer which are associated with a particular 

10 event marker. Fig. 4 shows that the annotations are associated with the event marker for 
"uses scissors." The annotations can then be presented in a video paper printout 432a as 
annotative information 414 associated with the group of icons corresponding to the 
annotated event marker. As mentioned above, the MFP 206 can perform a "cut" 
operation in a predefined area in an image of the barcode sheet where the user can make 

1 5 annotations. A subsequent "paste" operation can then be made to place the armotations 
into the video paper output 432a. 

[48] Fig. 4 illustrates an alternative embodiment in which the annotative area in the 
barcode sheet 404b is divided into N portions. In accordance with this embodiment, each 
portion (e.g., line of text) is an annotation that can be associated with an occurrence of the 
20 event corresponding to the marker. However, due to obvious physical limitations, there is 
only enough space for N occurrences of an event. In the barcode sheet 404b, assume that 
N is 3. The video paper might look like the output 432b. Where, the first three 
thumbnails 462 for the event "uses scissor" are annotated, but subsequent occurrences of 
the event are not annotated. 

25 

Timestamping the Information Stream 

[49] Referring again to Fig. 5, a step 506 is shown of "barcode association." An aspect 
of the present invention is the notion of "timestamping" an information stream with event 
markers. A fiirther aspect of the invention is that points in the information stream can be 
30 timestamped with the same event marker to indicate the occurrence of the "same" event. 
It is worth noting at this time, that the idea of "sameness" is defined by the user, and does 
not require any a priori determination or assessment by the system of the content of the 
information stream. As mentioned above, this allows a user to mark the information 
stream with "unassigned" markers to mark occurrences of unexpected events. In fact, the 

12 



notion of marking with voice-originated markers highlights an aspect of the invention 
wherein the markers themselves do not have to be predefined. 
[50] In the specific embodiment of Fig. 2, the event markers are barcodes, more 
particularly, suitable representations of the barcodes such as a unique number. However, 
S as discussed above, other "markers" can be used. In accordance with the present 
invention, markers are uniquely identified and some form of timing information is 
associated with each marker. This allows for markers to be time-wise correlated with the 
information stream. This creates information that associates each occurrence of a marker 
with a point in time in the information stream. Associating (step 506) the information 
10 stream with the markers in this way effectively "timestamps" the information stream with 
the markers. 

[51] Referring to Fig. 3 , for example, suppose a mark Mj had been timestamped at 
time ti; i.e., a first barcode was scanned at time ti, or a first phrase was spoken at time ti. 
Suppose fixrther that a mark M2 was timestamped at time ti, and that the mark Mi was 
1 5 again timestamped at time ts. In accordance with the invention, the mark Mi would be 
associated with the information stream at time ti (302a) and with the information stream 
at time ta (302b). Likewise, the mark M2 would be associated with the information 
stream at time t2 (304). 

[52] The association of a mark with one or more times can be represented in many 

20 ways. For example, in one embodiment, the mark-to-time association can have the 

following data structure: 

M,: 12:32:05, 12:35:22, 12:38:14 
M2: 12:34:19, and so on. 

25 Alternatively, the mark-to-time association can have the following data structure: 

M,: 12:32:05 
Mi: 12:35:22 
Mi: 12:38:14 
M2: 12:34:19, and so on. 

30 

One of ordinary skill will readily appreciate that still other representational structures can 
be used. 

[53] From the foregoing description, those of ordinary skill can readily embody this 
aspect of the invention in any of a number of ways. In a particular embodiment of the 
35 invention, the MFP 206 receives the information stream and a "stream" of timestamps 
(i.e., markers with suitable timing information), and correlates the information stream 
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with the timestamps on the fly, to produce for example a video paper 232 output. 
Alternatively, a table can be maintained that provides a pointer into the information 
stream file(s) for each timestamp. This internal representation would facilitate a video 
display presentation of the timestamped information stream, obviating the need for 
5 repeatedly re-indexing the information stream. 

[54] In accordance with a particular implementation of this aspect of the invention, a 
data structure is created for each marked event (timestamp). The data structure includes 
the id of the marker, the time of the event, a pointer to the information stream (media), 
and begin and end points for playback. The begin and end points can be modified at a 

1 0 later time even after the video paper has been printed. 

[55] As mentioned above, some form of timing information is associated with the 
markers. It can be appreciated that this information allows for the information stream to 
be synchronized with the markers. In accordance with one embodiment of the invention, 
absolute time can be used. Thus, the information stream can have an absolute time value 

1 5 associated with it. In the case of discrete media, such as images from a digital camera, 
each image can be associated with a time value. The recording device 202 might have a 
clock or other suitable time base for associating a time value with the information stream. 
[56] Similarly, the event marking device such as the barcode scanner 204a might 
include a clock so that for each marking event the device produces a datum representing 

20 the marker (e.g., a unique number representing a barcode, or digitized speech, etc.) and a 
datum representing the time at which the mark was made (e.g., scanned barcode, spoken 
speech). In this particular embodiment, the information stream and the set of markers 
with their corresponding time values collectively represent a timestamped information 
stream. This technique is applicable for both continuous and discrete information 

25 streams; especially so for discrete media since each "frame" is marked with a time value. 
The process of timestamping, then, amounts to matching the time values associated with 
each marker with the time values in the information stream. In practice, an exact match is 
not guaranteed due to clock skew between the time base in the event marker (e.g., 
barcode scanner) and the time base in the recorder. Thus, a margin of error should be 

30 provided; the amount of the error, whether milliseconds, seconds, minutes, etc., will 
depend on the nature of the subject being recorded. 

[57] A more appropriate technique for synchronizing continuous information streams 
might be to establish a temporal reference point where time can be measured relative to 
that reference point. For example, the information stream might include a reference point 
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at the beginning which can be used to "zero out" a timer in the event marking device 
(e.g., barcode scanner). A special signal can be inserted at the beginning of the media 
that the system (e.g., an MFP) can detect. Thus, a marking event that occurs five minutes 
after the timer was zeroed out would correspond to a point in the information stream that 
5 was five minutes into the information stream measured from the special signal. 

Additional reference points can be provided to periodically re-synch the clocks in the 
recorder and the event marking device to account for clock drift, if timing is critical. It 
can be appreciated of course that additional information would be needed to indicate 
occurrences of clock re-synchs. 

1 0 [58] Another technique for synchronizing an information stream to a series of event 
markers, is the use of START-STOP indicators. In this scenario, a barcode labeled 
START is scanned at approximately the same time as the recording device is started. 
When the barcode data and video are merged, the system recognizes the START indicator 
and maps the associated time to the beginning of the video. Thus, the offset from this 

1 5 time to a timestamp associated with another barcode is used to index into the video 
recording and pull out the appropriate clip. Optionally, a STOP code can be scanned 
when the recording is complete. This would be especially useful if several recording 
sessions are contained in a single information stream; e.g., on a single video tape. In that 
case, the portion of the information stream between a STOP indicator and the next 

20 START indicator can be factored out when indexing from the timestamp IDs to the video. 
[59] Alternatively, a "START marking event" indicator and a "STOP marking that 
event" indicator can be employed. Each event of interest can the be delimited by these 
pair of indicators. In addition, surrounding context for each delimited event can be 
provided by padding the beginning and/or end of each delimited event with a segment(s) 

25 of the information stream. The surroimding context can be provided, for example, during 
playback. 

[60] Fig. 5 shows a step 508 of "barcode storage." In accordance with one aspect of 
the invention, the timestamp information can be stored in the event marker device 104 
(Fig. 1) itself. Thus, in a particular embodiment of the invention, the barcode scanner 
30 might be configured with a suitable memory component to store the timestamps. The 
memory component can be a form of non-volatile memory if it is anticipated that the 
device might be turned on and off while collecting data. A disk drive unit might be 
provided. The timestamp information might include an encoding of the scanned barcode 
plus some timing information that indicates when each barcode was scanned. Additional 
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information might include metrics such as the number of times each barcode was 
scanned. 

[61] The barcode scanner (and more generally, an event marker device 104) can be 
configured to commiuiicate the stored timestamps to a suitable mass storage device for 
5 subsequent processing. Alternatively, the event marker device can be configured to 
transmit a timestamp as soon as it receives one, thus obviating the need to provide 
suitable storage for timestamps in the event marker device itself. 
[62] Typical processing in this step 508 includes: 

• receiving a marker such as a scaimed barcode, a selected graphic, a digitized 
10 spoken phrase, and so on, and converting the marker to a suitable internal 

representation (e.g., a unique number) 

• if the marker has not been previously received, then storing the marker and 
associating some timing information with the marker to indicate its time of 
receipt 

15 "if the marker has been previously received, then simply associate some timing 

information with the marker to indicate its time of receipt 

One of ordinary skill in the art can readily implement the foregoing in a suitable event 
marker device 104, such as a barcode scanner. 

20 

Manipulation of Timestamped Information Streams 

[63] Fig. 5 also shows a step 510 of "producing video paper." In Fig. 2, two 
embodiments of this aspect of the invention were discussed relating to accessing 
timestamped media. Video paper 232 can be used to access portions of the information 
25 stream by selecting a barcode 254 associated with one of the thumbnail images. Fig. 2 
also shows a video display device 234 for playing back a timestamped information 
stream. A video display device can offer the user a greater number of options and 
controls for accessing the information stream. 

[64] Fig. 5 shows a step 512 of "presenting." In accordance with this aspect of the 
30 present invention, the idea of "playback" comprises playing a segment of the information 
stream. For example, the media segment 3 12 shown in Fig. 3 is a segment associated 
with the timestamp Mi. Altematively, the information stream can be played back starting 
from a time based on the time of the selected timestamp. Fig. 3 shows that playback of 
the information stream can begin at a time 314 prior to the timestamp Mi, or at a time 316 
35 subsequent to the timestamp. 



16 



[65] It can be appreciated, however, that playback of a timestamped information 
stream can be initiated from the barcode sheet 204b (Fig. 2) itself, instead of initiating 
playback from the video paper printout 232. A barcode scanner can be configured to 
cooperate with a suitably configured playback device via a cable or a wireless connection 
5 to perform the desired playback frinctions. It is noted that the playback device and the 
record device can be one and the same. 

[66] The user can initiate playback by scanning the barcodes on the barcode sheet. 
Thus, a first "swipe" (scan) of the "uses teeth" barcode, for example, might cause 
playback of the first point in time in the information stream that is timestamped with the 
10 barcode corresponding to the event. A second swipe of the same barcode can cause 
playback of the information stream at the second timestamp, and so on. Continued 
swiping of the same barcode could eventually cause the information stream to "wrap" and 
start playing back from the first occurrence of the timestamp. 

[67] Alternatively, all clips associated with a barcode can be played back with a single 

15 swipe of the barcode. For example, when the user swipes the "uses teeth" barcode, each 
clip associated with that barcode can be played back. An appropriate delay between each 
clip can be provided as a transition from one clip to the next. 
[68] In accordance with yet and another embodiment, scanning of the ID simply 
displays a list of all clips associated with that ID. For example, one or more 

20 representative thumbnail images for each clip can be displayed. This could be usefril for 
playing back the clips off of the original barcode sheet in the context of any notes that the 
user has written down without having to print out any video paper. 
[69] In one embodiment of the invention, the event marking device 204a (Fig. 2) is a 
single button barcode scanner. Pushing the button activates the scanning operation and 

25 causes the device to read a barcode and store the corresponding ID along with timing 

information. When the device is coupled to an MFP 206 the ID's and timestamps are sent 
from the barcode scanner to the MFP and are used to timestamp the recorded sfream 
2 1 2a. Along with the timestamping information, the MFP includes the ID associated 
with each timestamp in the information for subsequent playback. The barcode scanner 

30 may be coupled to a suitably configured display system, such as described in one or more 
of the above referenced co-pending applications, for playback of the information stream. 
Activating the scanning operation causes the device to read a barcode and send the 
corresponding ID to the display system. Upon receiving the identifier, the display system 
locates one or more timestamps with that ID and displays the corresponding portions of 
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the recorded media. Receiving additional instances of the same ED causes the system to 
skip to the next corresponding timestamp. 

[70] It should be noted that the same device may be used both for timestamping an 
information stream associated with recorded events and displaying portions of the stream 
5 in response to received identifiers. In this case, the device (e.g., MFP) should have a 
method for switching between "association" mode and "playback" mode. This may be 
done by a control on the device or by sending a special "mode switch" identifier. 
[71] In an alternative embodiment, the barcode scanning device is equipped with two 
buttons, a RECORD button and a PLAY button. Pressing the RECORD button initiates a 

1 0 scanning action and subsequently the scarmed ID is saved to storage with the associated 
timing information as before. Pressing the PLAY button also initiates a scarming action. 
In addition, the device then searches its local memory for previously stored events with 
the same ID. If an event is found, the ID and the timing information associated with that 
event are sent to the display system. The scanning device may also store a 'PREVIOUS' 

1 5 pointer indicating which event was displayed for the ID. If more than one event is found 
and a 'PREVIOUS' pointer exists then the next event (e.g., with a later timestamp) for 
this ID is sent to the display system. If no events are found, the ID may be sent to the 
display system along with an indication that this is a playback request and cause the 
display system to locate and display the recorded stream as in other embodiments. 

20 [72] In accordance with another aspect of the invention, the "playback" action can be 
controlled by the use of task-specific markers. In a particular embodiment of this aspect 
of the invention, the playback-device might be a DVD player or an audio amplifier. 
Special "playback control" barcodes can be provided which might represent appropriate 
playback controls such as "play," "rewind," or "normal loudness." These barcodes might 

25 be stickers or some other marking method by which barcodes on a sheet of paper can be 
arranged to form a custom "remote control card". Thus, in the case of video paper, the 
printout might have playback control barcodes printed on it. If playback firom the original 
barcode sheet (e.g., 204b, Fig. 2) is desired, barcode stickers with predefined functionality 
can be affixed to the sheet to provide playback control. Alternatively, pre-printed sheets 

30 might also include pre-printed "playback control" barcodes. 

[73] For example, the video paper 232 of Fig. 2 can include control barcodes for 
operating the playback device by simply swiping the appropriate control barcode. This is 
convenient if a barcode scanner is already being used to playback timestamped media, 
where the media are stored in multiple playback devices. The user would not have to 
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constantly switch between the barcode reader and any one of a number of different 
remote controls to access her playback environment. Consider the very likely scenario 
where one or more TV monitors may be connected to multiple source feeds including one 
or more DVD players, a bank of CD players, video cassette recorders (VCR), and so on, 
5 where some of the devices might be daisy-chained so that video from the DVD might 
feed through the VCR, for example. Manipulating the plethora of remote controls to 
connect the desired source to the desired monitor can be a very trying experience. 
[74] However, if the equipment is provided with computer actuated switches, a video 
paper with appropriate control barcodes could greatly simplify the playback task. The 

10 user simply swipes the barcode corresponding to the desired action. The swiped barcode 
can then be detected by suitable control software to activate the equipment accordingly to 
connect the media playback device containing the accessed media to the user's display. 
Thus, for example, if a barcode corresponding to media on playback device A is swiped, 
then the control software can control the appropriate switches to connect playback device 

15 A to the user's display. If the user next swipes a barcode for which the associate media is 
on playback device B, the control software can connect together the user's display and 
playback device B. 

[75] In the case where multiple displays are provided, the video paper can include 
barcodes which indicate the display on which a selected timestamped media segment will 

20 be presented. For example, swiping a barcode associated with display device A can be an 
indication that all subsequent swipes of timestamped media will be presented on display 
device A, until the user swipes the barcode for a different display device. 
[76] The task specific makers might be sets of maker ID's commonly used for 
particular purposes like filming a "Birthday Party." All Birthday Party pages might have 

25 the same ID used for "blowing candles," "eating cake," etc. A special format for each 
task could be used in generating the printouts. The association between these special 
id's and various actions (e.g., the print format) could be made available on a website or 
other public repository. The ID itself could also code in the address for such information. 
[77] In accordance with another aspect of the invention, timestamping can be 

30 performed on an information stream that is being played back. The foregoing discussed 
aspects of the invention is applied to an information stream that is being recorded. That 
is, the information stream is being timestamped during the recording process. 
[78] However, the invention is equally applicable to a previously recorded information 
stream that is subsequently reviewed. As the user plays back an earlier recorded 
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information stream, she can use an event marker 102 (e.g., a barcode scanner) to mark 
events she observes during playback produce a stream of timestamps. These timestamps 
can then be synchronized with the previously recorded information stream in the same 
manner as described above. It can be appreciated, therefore, that the timestamping 
5 activity does not have to take place during the time the recording is made, but rather can 
be performed at a time subsequent to the recording. 

[79] Still another aspect of the invention is timestamping a previously timestamped 
information stream. During a review of the previously timestamped information stream, 
the user can use an event market to fiirther timestamp the information stream. Suppose a 
10 user is browsing a timestamped information stream, watching one minute clips, for 

example. The user might timestamp the clips themselves, to mark something of interest 
in the clip. More generally, any part of a previously timestamped information stream can 
be annotated with additional timestamps. 
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